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ABSTRACT 

This paper addresses the use of computer-based testing in distance education, based on 
the experience of Universitas Terbuka (UT), Indonesia. Computer-based testing has 
been developed at UT for reasons of meeting the specific needs of distance students as 
the following: 

> students' inability to sit for the scheduled test, 

> confIicting test schedules, and 

> students' flexibility to take examination to improve their grades. 

In 2004, UT initiated a pilot project in the development of system and program for 
computer-based testing method. Then in 2005 and 2006 tryouts in the use of computer- 
based testing methods were conducted in 7 Regional Offices that were considered as 
having sufficient supporting recourses. The results of the tryouts revealed that students 
were enthusiastic in taking computer-based tests and they expected that the test 
method would be provided by UT as alternative to the traditional paper and pencil test 
method. UT then implemented computer-based testing method in 6 and 12 Regional 
Offices in 2007 and 2008 respectively. 

The computer-based testing was administered in the city of the designated Regional 
Office and was supervised by the Regional Office staff. The development of the 
computer-based testing was initiated with conducting tests using computers in 
networked configuration. The system has been continually improved, and it currently 
uses devices linked to the internet or the World Wide Web. The construction of the test 
involves the generation and selection of the test items from the item bank collection of 
the UT Examination Center. Thus the combination of the selected items compromises the 
test specification. 

Currently UT has offered 250 courses involving the use of computer-based testing. 
Students expect that more courses are offered with computer-based testing in Regional 
Offices within easy access by students. 

Keywords: Distance education, online assessment; computer-based testing; testing 
methods, test delivery; examination 

INTRODUCTION 

Computer and Internet technologies are now widely used in the field of education to 
support teaching and learning activities, including assessment activities (Lee & Tsai, 
2005; Savery, 2002; Clariana & Wallace, 2002; Duran, 2000; Gibson et.al, 2000; Olgivie, 
et.al, 1999). Students now submit course work by email, complete learning activities 
through the World Wide Web, and complete student assessment in the form of online 
testing (Bishop, 2000; Newman, 2000; Bitzer, 2000). An Internet-based education 
environment facilitates student learning without the constraints of time and distance 
(Wang, 2007; Newman, 2000). Although Internet users still encounter some barriers. 
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such as a lack of skills, a lack of training, a lack of technologies to ensure authentication 
and security, and higher costs for accessing the Internet (Aojula et.al, 2006, Cassady & 
Gridley, 2005; Rowe, 2004; Tsai & Tsai, 2003; Jones, et. al, 2002; Eastin & Larose, 2000), 
the uses of the Internet in distance education have become common worldwide (Tsai, 
2005; Balasundaram & Ramadoss, 2005; Achtemeier et.al, 2003). 

Research on computer based assessments revealed that the students' attitudes toward 
the computer examinations in the course were very positive, the students comfort 
level of using computer increased, the students' reception to computer testing were 
positive, and the students preferred the web-based assessment to the paper and pencil 
assessment (Millet et.al, 2002; Olgivie et.al, 1999). Puhan et. al (2007), Poggio et. al 
(2005), and Alexander et.al (2001) found out that taking an exam online compared to 
traditional paper and pencil testing does not have effect on overall exam scores. 
However, the uses of computer based assessments in the testing administration require 
high levels of control and security in order to ensure the integrity of the testing results 
and process (Aojula et.al, 2006, Cassady & Gridley, 2005; Rowe, 2004; Jones, 2002; 
Morgan &0'Reilly, 1999). 

Universitas Terbuka (UT), Indonesia has developed the system and program for 
computer-based testing method since 2004, and has implemented this testing method 
since 2007, following successful tryouts in 2005 and 2006. Computer-based testing has 
been provided to meet the specific needs of UT students due to students' inability to sit 
for the scheduled test, conflicting test schedules, and students' opportunity to take 
examination to improve their grades. This paper addresses UT experience in developing 
the system and program for computer-based testing since 2004 to 2008, and conducting 
computer-based testing in 2007 and 2008. 

METHODS 

This paper describes UT experience in conducting computer-based testing, and presents 
results of the research in the area. The research method was descriptive using a case 
study design at one Regional Office conducting computer-based testing. Population and 
research sample of 30 students were the students who registered and took computer- 
based testing at one Regional Office in UT in academic year of 2008 semester 1. 

Data was collected between May and July 2008. Primary data were responses of the 
students to the questionnaire and information from proctor and Regional Office Staff 
during interviews, whereas secondary data were the student registration data, test 
scores of students taking computer-based testing and paper and pencil tests, and 
Regional Office report on the administration of the examination. The quantitative data 
were analyzed using the program SPSS 15.0 for Windows. 

RESULTS AND DISCUSSION 

Experience in Conducting Computer-Based Testing 

In 2004, UT initiated a pilot project in the development of system and program for 
computer-based testing method. Then in 2005 and 2006 tryouts in the use of computer- 
based testing method were conducted in 7 Regional Offices that were considered as 
having sufficient recourses. The results of the tryouts revealed that students were 
enthusiastic in taking computer-based tests and they expected that the test method 
would be provided by UT as alternative to the traditional paper and pencil test method. 

UT then implemented computer-based testing method in 6 and 12 Regional Offices in 
2007 and 2008 respectively. The computer-based testing was administered in the city of 
the designated Regional Office and was supervised by the Regional Office staff. The 
development of the computer-based testing was initiated with conducting tests using 
computers in networked configuration. The system has been continually improved, and it 
currently uses devices linked to the internet or the World Wide Web. The construction of 
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the test involves the generation and selection of the test items from the item bank 
collection of the UT Examination Center. Thus, the combination of the selected items 
compromises the test specification. Currently in 2009, UT has offered over 250 courses 
involving the use of computer-based testing. Students expect that more courses are 
offered with computer-based testing in more Regional Offices within easy access by 
students. 

Table: 1 presents the number of students taking the computer-based testing, the 
number of courses offered and registered, the number of online manuscripts 
downloaded during the period of the testing, and the number score for computer- 
based testing (CBT) and paper and pencil testing (PPT). Meanwhile, Table 2 presents 
comparison scores CBT and PPT from 2007.1 to 2008.2. 

Table: 1 

Number of Students Having CBT from 2007.1 to 2008. 2 


Year 

Semester 

N of 
R.O 

N of 

Students 

N of Course 

N of Online 

N of Score 

Offered 

Registered 

Manuscripts 

CBT 

PPT 

2007.1 

5 

41 

264 

66 

112 

112 

89 

2007. 2 

6 

56 

215 

67 

167 

167 

130 

2008. 1 

9 

79 

203 

85 

218 

218 

174 

2008. 2 

12 

125 

205 

111 

346 

346 

265 


Note: RO = Regional Office 


Table: 2 

Comparison Scores of CBT and PPT from 2007.1 to 2008.2 


Item 

2007.1 

2007.2 

2008.1 

2008.2 

CBT 

PPT 

CBT 

PPT 

CBT 

PPT 

CBT 

PPT 

N 

112 

89 

167 

130 

218 

174 

346 

265 

Mean 

46 

45 

48 

47 

50 

45 

54 

45 

Std. 

Deviation 

12.42 

10.77 

12.91 

11.32 

16.09 

11.54 

17.05 

12.06 

Minimum 

20 

24 

20 

23 

20 

22 

10 

10 

Maximum 

95 

74 

90 

83 

95 

75 

97 

78 


Note: 

CBT = Computer Based Testing; 
PPT = Paper and Pencil Test 


Case Study Research in 2008 Semester 1 

The case study was conducted in one Regional Office in 2008 semester 1 by distributing 
questionnaires to 30 students taking computer-based testing. The results of the study 
revealed the following information. UT website was the source of information about 
computer-based testing for about 65% of the students. The other sources of information 
were other fellow students and Regional Office staffs. The information was clear enough 
for most of the students (70%). About 70% of the students got information more than 
one month before the test, thus they still had the time to register for computer-based 
testing. 

The reasons for students taking this test were to get better grades (48% of the 
students), to have more than one examination in one semester (28% of students), to 
know the number of right answers, to have new experience in testing, and to manage 
schedule of the study. The students expected that they could arrange examination 
schedule, manage monitor self-studying, get flexibility, and have immediate feedback. 
In the other words, computer-based testing has been provided to meet the specific 
needs of UT students for the following reasons: (1) students' inability to sit for the 
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scheduled test, (2) conflicting test schedules, and (3) students' flexibility to take 
examination to improve their grades. 

The students reported that 47% of them registered and took computer-based testing for 
1 course, 37% of them for 3 courses, and remaining students for 2 and 4 courses. This 
test was the first time administered to about 60% of students. The students described 
that they had moderate ability in understanding and good enough skills in operating the 
computer. The students did not have handicaps with new technology. They perceived 
that the application for registration and testing was easy to use. Computer connection 
during registration and downloading of the test was good and fast. There were no failing 
hardware and electricity problems. The following table describes the number of students 
involved in this testing and the courses that was registered from 2007 semester 1 to 
2008 semester 2 at this designated Regional Office. 

Table 3 

The Number of Students Taking Test and Course Registration 


Year 

Semester 

Number of 
Students 

Number of 
Course 
Registered 

Average Ratio of 
Student-Course 

Number 

of 

Scores 

2007 

1 

38 

63 

2,68 

102 


2 

56 

65 

3,02 

169 

2008 

1 

79 

85 

2,76 

218 


2 

125 

111 

2,77 

346 


The Regional Office provided special room furnished with air condition system for 
computer-based testing in its Office. The computer-based testing was administered in 
the city of the designated Regional Office. Room for examination was good in terms of 
temperature, lighting, cleanliness, and tidiness. There were 15 sets of computers that 
met specification requirements. Those sets of computers were set up properly in terms 
of position of computers and monitors, space, and distance between tables of computers. 
Thus, the testees could comfortably do the test and the proctor could easily monitor the 
test administration. 

The students had enough technical assistance for operating the computer application, 
understanding testing procedure and exam regulation, and signing the attendance form, 
from the ICT and 3 other Regional Office staffs. The students reported that the 1CT and 
the other Regional Office staffs were ready to assist students and they did their tasks 
well. Thus, the computer-based testing was administered in the city of the designated 
Regional Office and was supervised properly by the Regional Office staff, as required by 
the examination procedures. 

UT used computer as media for objective test examination with supervision by proctors 
and assistance by ICT staffs. This support corresponded to the report about the success 
of computer use for various areas including learning and testing. Generally, the 
examination happened properly. There were 3 test sessions in every test day. However, 
20% of testees still needed clarification for registration procedure, confirmation 
procedure, and the date of examination, and 6 % of testees still needed clarification for 
test locations and method of paying the test fee. In order to increase performance 
quality of computer-based testing, the Regional Office should consider the following 
areas for improvement, such as providing more computers for testing uses, increasing 
the quota of testees, expanding the number of courses offered with computer-based 
testing, applying double sessions of the test per day, providing wider socialization about 
the test system and courses offered, clarifying method of registration and payment, and 
enhancing test confirmation service. 
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About 1 hour to 30 minutes prior to the test, proctors and ICT staff held the meeting to 
establish common perception about the procedure of the test and the mechanism to 
facilitate the testees. The proctors assisted the testees about how to fill attendance list 
and how to participate in this kind of testing. ICT staff installed the computer-based 
testing application and password to the Regional Office server and the computer clients. 

Then, he configured and set each computer properly and easily without any problem. The 
process of computer installation and configuration was going well. Time required for 
installation was about 30 seconds till 1 minute. Internet connection during item test 
generating was good and fast, thus the online manuscripts of the test could be easily 
downloaded. It seemed that ICT staff had enough time to re-preparation of the 
computer from one session to another session of testing. Time required for re-preparing 
the computer was about 5 to 10 minutes. The process of computer preparation and test 
item download could be correctly executed. Testees' answers were sent to UT Head 
Office everyday, right after the third session of the test finished through Internet 
connection through the UT website. The Regional Office also sent the CD backup of 
database of testees' answers after the period of examination finished in the Regional 
Office by using Post Office service. 

About 75% of respondents had both computer based testing and paper and pencil test. 
Table 2 and Figure 1 and Figure 2 provide information about respondents' scores of CBT 
and PPT. Pearson correlation between CBT and PPT scores was 0.37 and it was 
significant at the level of 0,01 and it meant that score of CBT had contribution to score of 
PPT about 14% {(0,37) 2 or 0,14}. Table 4 describes the frequency distribution of the 
two groups of scores. 


Table 4: 

Profile of CBT Score and PPT Score 


Items 

N 

Mean 

Standard 

Deviation 

Minimum 

Maximu 

m 

Score of CBT 

93 

54 

18. 83 

23 

95 

Score of PPT 

71 

46 

11. 55 

22 

75 



Figure 1. Scores of CBT 


Figure 2. Scores of PPT 
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Table: 5 

Frequency Distribution of CBT and PPT Scores 


No. 

Score 

CBT Frequency 

% 

PPT Frequency 

% 

1 

40 

25 

26.88 

25 

34.21 

2 

41 - 50 

24 

25. 81 

25 

35.21 

3 

51 - 60 

13 

13. 98 

15 

21.13 

4 

61 - 70 

7 

7.53 

3 

4.23 

5 

71 - 80 

12 

12. 90 

3 

4.23 

6 

> 80.00 

12 

12.91 




CONCLUSIONS 

Students were enthusiastic in taking computer-based testing and they expected that the 
test method would be provided by UT as alternative to the traditional paper and pencil 
test method. The number of students taking CBT and the number of courses with CBT 
registration increased from semester to semester. The students expected that they could 
arrange examination schedule, manage and monitor themselves in their study, get 
flexibility, and have immediate feedback. Thus, computer-based testing has been 
provided to meet the specific needs of UT students in the following areas: (1) students' 
inability to sit for the scheduled test, (2) conflicting test schedules, and (3) students' 
flexibility to take examination to improve their grades. 

Students expected that more courses were offered with computer-based testing in more 
Regional Offices within easy access by students, and more computers were provided for 
testing. They also expected that UT could increase the quota of testees and design more 
sessions of CBT, In addition, they also still needed more assistance for procedure of 
registration and test confirmation, and mechanism of getting test information and paying 
the registration fee. 

The computer-based testing was administered in the city of the designated Regional 
Office and was supervised by the Regional Office staff. The students had enough 
technical assistance for operating the computer application, understanding testing 
procedure and exam regulation, and signing the attendance form provided by the 
Regional Office staffs. 

UT had utilized computer as media for objective test examination with supervision by 
proctors and assistance by ICT staffs. The development of the computer-based testing 
was initiated with conducting tests using computers in networked configuration. The 
system has been continually improved, and it currently uses devices linked to the 
internet or the World Wide Web. 

The construction of the test involves the generation and selection of the test items from 
the item bank collection of the UT Examination Center. 
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